Skip to content

fix(s3): stream wr.s3.download in chunks instead of loading whole object into memory - #3396

Closed
nileshpatil6 wants to merge 1 commit into
aws:mainfrom
nileshpatil6:fix/s3-download-memory
Closed

fix(s3): stream wr.s3.download in chunks instead of loading whole object into memory#3396
nileshpatil6 wants to merge 1 commit into
aws:mainfrom
nileshpatil6:fix/s3-download-memory

Conversation

@nileshpatil6

Copy link
Copy Markdown

Closes/relates to #2831.

wr.s3.download currently opens the object with s3_block_size=-1 (one shot download), which fetches the entire object into an in memory cache inside _S3ObjectBase, then calls s3_f.read() to pull the whole thing out as a second bytes object, then writes it to the local file. For large files this means the object briefly exists twice in memory before it even reaches disk, so downloading a multi GB file can OOM a process that would otherwise have plenty of headroom for a streamed copy.

This changes download() to open the object with a fixed 8 MB block size instead of one shot, and copies it to the destination in a simple read/write loop (_copy_in_chunks) instead of a single read() call. That keeps memory usage roughly proportional to the block size regardless of the source object size. The underlying _S3ObjectBase.read() already supports incremental ranged reads when s3_block_size is a positive number (this is the same mechanism _read_text_core.py already uses for streaming reads), so this reuses existing, tested code paths rather than adding new S3 fetch logic.

Added test_download_file_chunked in tests/unit/test_moto.py, which monkeypatches the block size down to 5 bytes and downloads an object whose size is not an exact multiple of the block size, to exercise the multi chunk path and confirm the bytes are reassembled correctly.

Testing done locally:

  • ran the existing test_download_file and test_download_fileobj tests plus the new test_download_file_chunked against moto, all pass
  • ran the full tests/unit/test_moto.py suite, 45 passed / 1 failed (test_glue_get_partition, unrelated pre-existing failure caused by a missing pyparsing dependency for moto's glue mock in my environment, not related to this change)
  • ruff check and ruff format --check pass on the touched files

I did not have access to real AWS infra so I couldn't run the basic/full test environments described in CONTRIBUTING.md, only the mocked moto based unit tests.

@nileshpatil6 nileshpatil6 closed this by deleting the head repository Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant